S3Drive - Storj Rclone upload missing ETag

Scenario: 1) Data is stored on a Linux machine and synced to S3 bucket with encryption 2) iOS file tries to open PDF 3) Opening PDF fails with "File preview not available" Same happens on macOS (edited)

You've said: "rclone 0.65" https://discord.com/channels/1069654792902815845/1069654792902815848/1159811026582241290, are you sure version number is correct? https://rclone.org/changelog/ You've mentioned that problem has gone after clearing thumbnail cache. Is that correct? If that's the case the issue is somewhere around caching with the app when file preview runs. We're going to check if we can improve that (e.g. corrupted cache/local file can be detected automatically).

Sorry, rclone v1.64.0 Clearing the cache worked in a previous version, in the current version doesn't help... (edited)

Are you using filename encryption? Does the name get correctly decrypted? Can you try opening a file externally: "Open with" on iOS? It would be good to rule out a possible decryption issue. If you go to "Logs" (long-tap version number on about page), is there any error there? (edited)

Yes, I use filename and directory encryption. I can see the files and directories, that works. Open With does not work as well

11:47 AM

Problem so far only with PDFs... TXT and JPEG works

Did you manage to check the logs? If file can't be opened externally than either this process fail decryption file, or there is a permission issue with the tmp directory on iOS or the PDF format isn't right. There should be something in logs if there is a failure for the first two items. (edited)

how can I open the logs? Long pressing "About" where on iOS? Same problem is on MacOS, there the logs are empty (edited)

Tom

Did you manage to check the logs? If file can't be opened externally than either this process fail decryption file, or there is a permission issue with the tmp directory on iOS or the PDF format isn't right. There should be something in logs if there is a failure for the first two items. (edited)

TXT and JPEG works, only PDF fails...strange

jggcvghhhh

how can I open the logs? Long pressing "About" where on iOS? Same problem is on MacOS, there the logs are empty (edited)

It's the drawer menu that you open with top-left icon, then there is "About" menu item. Once you open About page, then at the very bottom there is a version number that you need to long-tap in order to open Logs.

ah.... #0 Preview failed:.... Null check operator used on a null value, status 500

jggcvghhhh

ah.... #0 Preview failed:.... Null check operator used on a null value, status 500

Right, at least tells us that app failed at some point. Let me check that.

jggcvghhhh

how can I open the logs? Long pressing "About" where on iOS? Same problem is on MacOS, there the logs are empty (edited)

On macOS, can you try downloading a file? Does it appear in Download folder? Does it have non-zero size? Does it open externally? (edited)

12:14 PM

Download does not work, open externally does not work as well (edited)

12:20 PM

This is my rclone config [s3drive] type = crypt filename_encoding = base64 remote = storj:s3drive3333 password = <password> filename_encryption = standard directory_name_encryption = true suffix = none storj is configured separately with standard parameters, no suffix, encoding etc.

I will get back to you. There is some error that occurs during file fetch, but we actually fail to capture it properly hence the: "Null check..." issue.

1

We've prepared 1.5.7 macOS release which improves error handling in that area and adds new feature: "Show info" which can be used to display object S3 raw data including it's headers. It likely won't fix your download/preview issue, but will likely help us to narrow down where the issue is. It's available in pre-release: https://github.com/s3drive/app/releases/tag/1.5.7, feel free to install it now and will be released ~early next week. Please also attach "Show info" output. This is available in the file context menu at the bottom, after enabling it in the settings.

Install the version, turned on object info and when I try to get the object info, the following screen appears

7:20 AM

7:21 AM

If I upload a PDF via S3drive app, opening it afterwards works

Thanks for letting me know, ~~it seems that Storj uses different date format~~ (it's actually related to user device locale settings, nothing to do with Storj in the response) we'll have it resolved. Since we've also improved logging, can you please try previewing/downloading PDF and check the logs? Previously there was: "Null check..." issue, but now there is a chance there will be more details to it. (edited)

In the meantime we've also updated the macOS build to hopefully resolve the FormatException issue with: "Show info": https://github.com/s3drive/app/releases/tag/1.5.7

Object Info..

Perfect ! That explains a bit. There is no valid ETag coming from the S3 API. Is that Storj?

12:33 PM

BTW. If thats Storj, are you running Storj gateway yourself or use their public endpoint? (edited)

I use the public endpoint with storj (edited)

12:34 PM

https://gateway.storjshare.io

12:34 PM

I try with B2 and let you know

I tried the same scenario with backblaze, and it works out of the box. so it really seems to be related to storj

6:02 PM

Same config as with storj (edited)

Thanks for letting me know @jggcvghhhh Hi @storj-derek, sorry for reaching to you directly, but I've thought you could shed some light on this issue and help us to resolve it. In summary, in some cases Storj doesn't return ETag. I don't know any other S3 API which does this. Basically ETag is always expected to be present. In some cases (e.g. multipart upload) it might have different format, but it is never empty/null. Is this issue some specific design of Storj S3 Gateway or perhaps is this a bug? I've found similar issue here, but apparently it's now resolved: https://github.com/storj/storj/issues/3761 (edited)

AWS Gateway: enable ETag · Issue #3761 · storj/storj

It appears that the gateway does not send the ETag. This causes S3DataError: BotoClientError: ETag from S3 did not match computed MD5 messages on file upload by the python-boto clients such as dupl...

8:51 AM

@jggcvghhhh Just a question, does it specifically happen for: ".pdf" files uploaded via Rclone? In other words if you upload: ".txt" file via Rclone does Etag gets correctly generated? (edited)

Yes, txt is fine and can open it. PDF fails

I won't be able to solve issues on the Storj level, but I've tried to set up Rclone with crypt in order to reproduce the request that leads to an object without valid ETag on the Storj side. I am using rclone v1.63.0 and configured my storage like:

[storj]
type = s3
provider = Other
access_key_id = <redacted>
secret_access_key = <redacted>
endpoint = https://gateway.storjshare.io

[storj_crypt]
type = crypt
filename_encoding = base64
remote = storj:my-photos
password = <redacted>
filename_encryption = standard
directory_name_encryption = true
suffix = none

I've then copied test pdf like this: rclone copy test.pdf storj_crypt: and I get valid object with ETag and openable within S3Drive. Can you post your Rclone storj remote configuration? You've initially posted config, but that's just crypt: https://discord.com/channels/1069654792902815845/1159814485515710525/1159827592732479590

I use type = storj #instead of type=s3 Try that tomorrow. Thanks a lot for your efforts! (edited)

jggcvghhhh

I use type = storj #instead of type=s3 Try that tomorrow. Thanks a lot for your efforts! (edited)

Glad I can help. Based on this comparison: https://rclone.org/storj/ you may want to benefit from S3 back-end anyway.

Use the s3 backend and one of the S3 compatible Hosted Gateways to increase upload performance and reduce the load on your systems and network. Uploads will be encrypted and erasure-coded server-side, thus a 1GB upload will result in only in 1GB of data being uploaded to storage nodes across the network.

At the bottom there is a limitation of Storj backend

Limitations: Storj backend: rclone checksum is not possible without download, as checksum metadata is not calculated during upload

I don't know much about Storj, but perhaps this would explain the ETag issues. In other words, when uploading file through Storj (non-S3) API the hash is never calculated. If you upload it through S3 gateway, then MD5 (aka ETag) gets calculated (stored as metadata in Storj) and then retrieved through S3 API. This table: https://rclone.org/overview/ specifically shows thay Storj has no hashing, whereas: Amazon S3 (or S3 compatible) has: MD5.

2

Tom changed the channel name: Storj rclone upload missing ETag - can't open/download files 10/10/2023 10:17 AM

Tom changed the channel name: Storj Rclone upload missing ETag - can't open/download files 11/11/2023 7:10 PM